Combining audio and gestures for a Real-Time Improviser

Skilled improvisers are able to shape in real time a music discourse by continuously modulating pitch, rhythm, tempo and loudness to communicate high level information such as musical structures and emotion. Interaction between musicians, correspond to their cultural background, subjective reaction around the generated material and their capabilities to resolve in their own terms the aesthetics of the resultant pieces. In this paper we introduce GRI, an environment which incorporates music and movement gestures from an improviser to acquire precise data and react in a similar way as an improviser. GRI takes music samples from a particular improviser and learns classifiers to identify different improvisation styles. It then learns for each style a probabilistic transition automaton that considers gestures to predict the most probable next state of the musician. The current musical note, the predicted next state, and gesture information are used to produce adequate responses in real-time. The system is demonstrated with a flutist equipped accelerometers and gyros to detect gestures. The results are very promising.